End user mobile task automation using multimodal programming by demonstration

Conversational agents are often used to perform tasks on smartphones, but existing conversational agents are limited in capabilities and lack of customizability. My work explores using the programming-by-demonstration approach to enable end users to program new tasks for conversational agents by demonstrating using the familiar graphical user interfaces of third-party apps. I propose to use a multi-modal (demonstration and verbal instruction) interface to support generalization, editing, error handling as well as creating control structures in creating such smartphone automation.